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ABSTRACT 



A recent study at California State University at Northridge 
reported that students in a virtual classroom score 20% higher than students 
in a traditional classroom. The primary purpose of the study presented in 
this paper was to replicate these results and to investigate advantages and 
disadvantages of a World Wide Web-based class. Major objectives were to: (1) 

evaluate performance differences between students in a Web-based class and 
those in a traditional classroom setting when students are allowed to select 
the course they prefer; (2) evaluate the hypothesis that the background of 
students who prefer to enroll in a Web-based course is not different from 
those who prefer to enroll in a traditional class; and (3) identify what 
advantages the technology used in a Web-based class has in terms of enhancing 
learning objectives. Participants were 89 students enrolled in two sections 
of an introductory statistics course at Texas A&M University Corpus Christi, 
one taught in a traditional classroom and the other taught entirely via the 
Internet. Dependent variables were student demographics, test scores, and 
student withdrawal and passing rates. The major conclusion was that the 
results reported in the earlier study do not appear to be reproducible, at 
least with this student population. (Contains five tables and six 
references.) (AEF) 
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S^rumenrd"noir«ssl% Abstract: A study at California State University at Northridge, (Schutte, 1997) and INFORMATION CENTER (ERIC) 

official OER I position cr policy. (Schutte, 1998), reported that students in a virtual classroom scored 20% higher than 

students in a traditional classroom (p<0.001). This paper describes the results of an 
investigation into the demographic and performance differences between students in an 
all web-based class in introductory statistics to students in a traditional classroom setting. 

It also presents some conclusions, recommendations and raises some questions requiring 
further study. 



Introduction 

A study at California State University at Northridge, (Schutte, 1997), (Black, 1997) & (Schutte, 1998), 
reported that students in a virtual classroom scored 20% higher than students in a traditional classroom 
(p<0.001). Professors at the University of Kentucky, reported a decrease in the failure rate of freshmen 
chemistry students using their web-based practice quizzes, (see Kiser & Toreki, 1997). Their work 
suggests that a web-based course could improve student retention and performance, at least in 
undergraduate mathematics and science courses. 

Schutte speculates that the reason for the higher performance of his students in the web-based class is 
increased group interaction among students in a web-based class. Although he did not present data to 
support this finding, he concluded: 

"The number one residual finding of my study is that students placed in a virtual 
environment, performing the same work as classroom -based students, are more likely to 
engage others for help and understanding (not to be confused with collaboration or group 
tasking, as is done in Neal's review) than are students placed in a face-to-face 
environment where constant contact is the norm." 

(Schutte, J. G. 1998) 

Ed Neal and others have criticized the research design and methodology of Schutte. Neal writes: 

"Unfortunately, Schutte ’s research design and methodology are so flawed that the results 
of the study are uninterpretable." 

(Neal, E. 1998) 

(Schutte, 1998), counters the criticism of Neal and others. It appears that much of the criticism and debate 
may be precipitated by a general lack of rigorous, well-designed studies of web-based learning. The 
primary purpose of this study was to attempt to replicate the results reported by Schutte and to investigate 
the advantages and disadvantages of a web-based class. The major objectives can be summarized as: 

1. To evaluate the performance differences between students in an all web-based class and those in a 
traditional classroom setting when students are allowed to select the course they prefer, 

2. To evaluate the hypothesis that the background of students who prefer to enroll in an all web- 
based course is not different from those who prefer to enroll in a traditional class, and 

3. To identify what advantages, if any, the technology used in a web-based class enhances the 
learning objectives of an introductory statistics course. 
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Methodology 



Since, the experiment described by Schutte detailed a remarkable claim of success by the virtual classroom, 
a similar, but larger, study has been conducted at Texas A&M University Corpus Christi with a similar 
protocol. During the Fall of 1997, 89 students enrolled into two sections of an introductory statistics 
course, one taught in a traditional classroom and the other taught entirely via the Internet. 

Similar to the Schutte study, this was an undergraduate statistics class. Likewise, the web-based class used 
the same general means employed by Schutte - lecture notes, on-line quizzes, a discussion listserver, email 
and biweekly live web classes (chat sessions). (Schutte, 1998) reported that his server crashed 14 times 
during the semester. In this study, the server was only off-line for 4 hours during a scheduled power 
outage. In addition, the lecture notes over the more difficult material were served as a PowerPoint lecture 
with synchronized audio; Real Audio compressed to server over a 28.8k modem connection (visit 
http://stat.tamucc.edu/m2342/chapl2/xrchart/ for an example lecture). 

However, unlike the Schutte study, the students were not randomly divided between the classes. Instead, 
students registered for the class of their choosing during normal registration. This was done since: 

1 . an objective of this study was to evaluate differences in demographics between students who 
prefer to enroll in a web-based class arid those that do not, and 

2. it was felt that it would be unfair to force students into the web-based class since it required 
some familiarity with the use of a web-browser and convenient access to the Internet. 

To account for the non-random assignment of students to these classes, the overall university GPA of all 
students was recorded and all students completed test A/4F distributed by the Mathematical Association of 
America on college-level algebra, the only prerequisite for the course. This same test is administered every 
semester to incoming freshman, providing a large database of information about the average scores on this 
test and how they correlate with student performance in introductory statistics and other mathematics 
courses. 

In addition, both classes ran concurrently for 15 weeks and were taught by the same instructor. Both 
covered the same content, had the same homework assignments, used the same text. Elementary Statistics 
by (Triola, 1997), and followed the same syllabus (visit http://stat.tamucc.edu/classes/m2342.html for the 
syllabus). Grades were based upon the same three written in-class, multiple-choice examinations 
administered at weeks 5, 10 and 15. The week 15 exam was the final exam which was a 2.5 hour 
comprehensive examination. Both classes took these examinations under the same conditions, and student 
names were obscured during grading to minimize testing bias. 

The primary dependent variables under analysis were: 

1. Student demographics - age, GPA, completed hours, and pretest scores 

2. Scores on the three, in-class, multiple-choice in-class examinations 

3. Student withdrawal and passing rates 

Student demographics (1) were compared using the Wilcoxon test. Examination scores (2) were compared 
using the repeated measures analysis of covariance. This analysis was conducted both with and without the 
use of the covariates - GPA and algebra pretest. Withdrawal rates (3) were compared using Fisher’s exact 
test. All calculations were done using the Statistical Package for the Social Sciences, SPSS version 8.0 
(visit http://www.spss.com for a description), and all were conducted using a significance level of 0.05. 



Comparison of Student Demographics 

As (Table 1) indicates, the students who enrolled in the all web-based class were somewhat more mature 
students. The web-based class was statistically significantly older than the traditional class (p<0.001). 
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Likewise, they had a higher GPA and had completed more university hours. However, the differences in 
GPA and university hours for enrolling students were not statistically significant. Interestingly, although 
the students in the web-based class had an average GPA about 7% higher than the traditional class, their 
algebra pretest scores were only about 2 % higher. 





Number of 
Students 


Avg. Algebra 
Pretest 


Avg. GPA 


Avg. Univ. Hours 
Completed 


Avg. Age 


Web-Based Class 


33 


18.7 


2.93 


80.5 


29.6 


Traditional Class 


56 


18.3 


2.72 


68.6 


22.9 


Wilcoxon p-value 




0.891 


0.081 


0.079 


<0.001 



Table 1: A comparison of demographics for enrolling students. 

Comparison of Examination Scores 

The following figure depicts the average scores between the two classes for the three major examinations. 
On average, the web-based class did score higher than the traditional class. Notice, however, that the 
difference between the two classes appears to narrow as the semester proceeded. 
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Figure 1: Average Unadjusted Examination Scores 



Source 


DfH 


DfE 


F-test 


p-value 


Exams 


2 


66 


7.99 


<0.001 


Classes 


1 


67 


7.14 


0.009 


Exams by Classes 


2 


66 


1.28 


0.289 



Table 2: Repeated measures analysis of variance (ANOVA) of examination scores. 

The repeated measures ANOVA, (see Table 2), of the examination scores indicates that the difference in 
examination scores is statistically significant (p=0.009). However, these differences can be explained by 
the higher average GPA and pretest scores of the web-based class. 

(Table 3) gives the same analysis using the pretest scores and student GPA as covariates. A graphical 
display of the least squares means from the adjusted analysis are displayed in (Figure 2) and (Figure 3). 
When the algebra pretest score is taken into account, the differences between the two classes narrowed to 
only 3% for the final examination, and the difference is no longer statistically significant (p=0.096). This is 
in spite of the feet that the difference in pretest scores among students finishing the course is not 
statistically significant (p=0.812), (see Table 5). 

The difference between the two classes is even smaller when student GPA is taken into account (p=0.390), 
(see Figure 3). However, among students who completed the course, the web-based class did have a 
higher mean GPA (p=0.016), (see Table 5). Note that not only does the difference in adjusted scores 



narrow as the semester proceeded, by the final exam, the average adjusted score for the traditional class 
was higher than the web-based class. 



Source 


p-value adjusting for 
algebra pretest scores 


p-value adjusting for 
student GPA 


Exams 


0.023 


0.066 


Classes 


0.096 


0.390 


Exams by Classes 


0.379 


0.152 


Pretest 


<0.001 


<0.001 



Table 3: Repeated measures Analysis of Covariance of examination scores. 




Figure 2: Average examination scores, after adjusting for pretest scores 




Figure 3; Average examination scores, after adjusting for student grade point average (GPA) 
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Comparison of Withdrawal Rates 



At Texas A&M University Corpus Christi, students are allowed to withdraw from any course up to the last 
day of class without the instructors approval and without receiving a grade for the course on their 
transcript. It is not unusual to have a student who is passing a course withdraw from the course in the final 
weeks of the semester because of unforeseen personal circumstances or because they needed an "A" in the 
course to get into a medical school or a highly selective graduate program. Additionally, this course is a 
service course to science majors. That is, everyone taking this course is doing so because it is a required 
course outside the core courses of their particular major. This creates an environment with a higher than 
desired student withdrawal rate and a lower than desired passing rate. 





Total 

Students 


Withdrew 

from 

Class 


Percent 

Withdrawing 


Number with 
Grade of C or 
Better 


Percent with Grade of 
C or Better 


Web-Based Class 


33 


9 


27% 


20 


60% 


Traditional Class 


56 


10 


18% 


37 


66% 


Fisher's p-value 






0.301 




0.652 



Table 4: Comparison of withdrawal and passing rates. 

The withdrawal and passing rates for the two classes are summarized in the above table. Although the 
differences in withdrawal and passing rates are not statistically significant, the web-based class did. 
experience a higher withdrawal rate and a lovver passing rate than the traditional class. 





Students 

Completing Course 


Avg. Algebra 
Pretest 


Avg. GPA 


Avg. Univ. Hours 
Completed 


Avg. Age 


Web-BasedClass 


24 


19.1 


3.04 


75.3 


30.7 


Traditional Class 


46 


18.4 


2.73 


68.7 


22.8 


Wilcoxon p-value 




0.812 


0.016 


0.344 


<0.001 



Table 5: A comparison of student demographics for students completing course using the Wilcoxon test. 

The above table, (Table 5), describes that analysis of the demographics for students completing the course. 
Notice that although the average pretest scores and average university hours completed are still not 
statistically significantly different between the classes (p=0.892 & p=0.344 respectively), the average GPA 
of students completing the web-based class was about 1 1% higher than the traditional class (p=0.016). 

Summary and Conclusions 

(Schutte, 1997) reported that the students in his web-based class scored 20% higher than his traditional 
class. In this study, the difference between average scores on the final exam is only 6 points higher in the 
web-based class. Similar to the Schutte study, no statistically significant differences in GPA were seen 
among the students enrolling in the two sections (p=0.081). There was an even smaller difference between 
pretest scores (p=0.891). However, when the examination scores are adjusted for either the pretest scores 
or student GPA, differences in the average examination scores are no longer statistically significantly 
different (p=0.379 & p=0.390 respectively). Hence one would conclude that the higher average GPA of the 
students completing the web-based class could explain the 6% difference in the average score on the final 
exam. 

This raises the question of why this study did not replicate the results of Schutte. It is unlikely that the lack 
of complete randomization in this study accounts for the different results since the students in the web- 
based class had better preparation, as measured by their pretest scores, GPA and number of completed 
university hours. This would have predicted an even larger advantage for the web-based class than that 
reported by Schutte. 
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Schutte also claims that students in a web-based class are more likely to “engage others for help and 
understanding.” This was not seen in this study. In fact, every attempt to organize the web-based class 
around a team project and introduce them to one another was strongly resisted. Students complained that 
the reason they enrolled in the web-based class was for schedule flexibility and that it was too hard to 
coordinate schedules between students to meet as a team. Likewise participation in the class discussions on 
the listserver and the voluntary weekly live web-classes was poor. On average, only 15%-20% of students 
attended the live web-classes, although they were loyal and enthusiastic toward these classes. 

It could be that the students in Schutte ’s classes were more motivated since his course was probably not a 
service course. Likewise, his students may have been younger and less involved raising families and 
working. In this study, the average age of the students completing the web-based class was 30.7. On 
average they were enrolled in 1 1.8 semester hours and were employed 22.3 hours per week. It can also be 
speculated that Schutte's teaching style may be more suited to a web-based class than a traditional class. 

The major conclusion of this study is that the results reported by (Schutte, 1997) do not appear to be 
reproducible, at least with our student population. This also points to the need for further careful research 
into the best teaching methodology for web-based classes. It is unlikely that a simple translation of a 
course from a traditional classroom format to an all web-based class will improve or degrade student 
learning performance. 

However, at Texas A&M Univ. Corpus Christi, we are continuing to offer web-based classes since they 
offer an alternative learning format that seems to be increasingly popular with a segment of our student 
population. Like many universities, we will continue to explore new teaching methodology for web-based 
classes. 
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